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ABSTRACT 

This manual is a beginning guide to program 
evaluation. Designed for program evaluators who have little or no 
previous experience in manipulating data^ this guide is designed to 
demonstrate how to evaluate different aspects of a vocational 
rehabilitation program — and use those findings for program 
improvement. The use of averages, percentages, and computers is 
discussed in relation to vocational rehabilitation data 
classifications. A separate section on the reporting of program 
evaluation results is also included in this practical guide. (SJL) 
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PREFACE 

"Program Evaluation: A Beginning Statement" was the product of a 
prime study group's consideration of program evaluation (Tenth Institute 
on Rehabilitation Services - 1972). 

As a result of that publication, the West Virginia Research and Training 
Center was requested to prepare a beginning guide to program evaluation 

This publication is that guide and is designed for program evalu^tors who 
have little or no previous experience in manipulating data. If you fall into 
this category, we hope the document will bridge the gap and be a beginning 
for you. Later, you may want to build your skills in the area of statistical 
concepts; but for now you have a more basic job that of learning how 
to start evaluating programs. We hope this document helps. 

If you are somewhat skilled in program evaluation, we hope you find some- 
thing useful here. 

If you are a skilled evaluator, put this document on your shelf or give it 
away. If you dare to read it, remember that not everyone is knowledge- 
able in the area of sophisticated PE (Program Evaluation) techniques and 
.they have to begin somewhere. 
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CHAPTER 1 



INTRODUCTION 

Does the term "Program Evaluation (PE)" scare you ' Did you know 



odology of researchers involving such things as control groups, independ 
and dependent variables, longitudinal studies, and various sampling tech- 



Many kinds of PE are not complicated and require little or no background 
in research methodology. Other types of PE require only that the evalua- 
tor understand the concepts involved and know where to go to find assis- 



The idea that PE has to be difficult to be good is wrong. For example, 
some relatively simple work with averages or percentages using available 
data can provide a wealth of information. This -is the least complicated 
type of PE. In addition, a basic understanding of the uses of a lew statis- 
tical techniques can extend the range of types of PE that you can do. 

This guide is designed to show you how to evaluate ditfcrcnt aspects ol 
a VR program - and use those findings for program improvement The 
approach is based on common sense, the procedures arc smiplc. and whci 
reference is made to a statistical concept, it is explained in common scns( 
terms rather than being overly technical. 



After you have used some bf these basic kinds of PE, you will be ready t^ 
consider more involved techniques. But for now, we'll stick to the easier 
kinds of PE, and show how you can evaluate using currently existing date 




niques. 



tance. 



CHAPTER 2 



GETTING READY FOR PROGRAM EVALUATION 

Before we get into the **nitty gritty" of program evaluation (PE), lct\ 
take a walk through the system we want to evaluate. 

STEP 1 : 

What is the total process that a client might go through in a 
vocational rehabilitation program? 

Figure 1 shows that process. As you will note, each step is coded with 
status numbers currently in use by VR agencies. 

Does the process look familiar? Of course! We have included this just 
to help you see the process on paper. The coded numbers and titles 
represent the entry point of a client, the kinds of services which may be 
provided, and the client's possible exit points. These different steps 
provide a basis for beginning your PE program. 

STEP 2: 

Where can you get data related to the system? 

The simplest place is the SRS-RSA-300 (R-300) form (see Figure 2). 
It - or its equivalent - is completed on all VR clients. ' 

STEP 3: 

Get a copy of Chapter 1 3 of the VR Manual referred to on the 
back of the R-300 since it explains how some of the information 
is to be placed on the form. 

If you have studied the system, reviewed the R-300 or its equivalent, 
and have Chapter 1 3 of the VR Manual, you have the basics to start 
program evaluation. Sound easy? It is! We'll show you what we mean 
in our next chapter. 



i 



-4- 










O \ 1 


o 


1 / 




o 


k-( > 














> 


a 




n 






7> 


•n 




2 


n 




0 


a 




r 


a 
> 




m 


r 




0 


1/) 




0 


H 




m 


> 




APP 


TUS 




r 






n 






> 






H 






0 






Z 






(/) 







o 

> 
o 

> 
o 

•n 

O 
r 

m 
2 



2 
< 

■0 

O 
O 
3) 
> 
2 



Q 
G 

W 




ERIC 



-5- 



Form RSA-300 
Revived 7 C 

Agency Code _ 



I' S DEPARTMENT OF HEALTH, EDUCATION, AND WELFARE 
National Center for Social Statistics 
48 WasJungton,D.C. 20201 

Case Service Report: Federal-State Program of Vocational Rehabilitation 

PART 1 (TO BE RECORDEO AT TIME OF FIRST REFERRAXI 



Form Approved 
Budget Bureau 
No 83-R0O40 



A. Last Name 



First Name 



Initial 



B. Address: Street and Number 

T 



City 



County 



Code^ 



Zip Code 



C. Refenal Date 

E. Age DOB_ 



D. Referral Source 



Sex: 1 □ Male; 2 □ F-emale 



G. Disability as Reported (describe) 



( mle , 



A. Soc. Sec. Nr. 

B. SSDl Status at Referral ^ 

C. Race ^ 

D. Date Referral Process Completed 

E. Months m Statuses 00-02 

F. Spanish Surname Yes □ !;Non 2 

G. Outcome of Referral Process: 

NOT ACCEPTED: Reason * 

I □ from (00); 2 □ from (02)' 

Client Referred to * 

OR 

Referral Not Appropriate- — □ 00 

If cfottd from SUtu$ 00, compltU lttin$ 
aA through 3C. If cloMd from Status 02, 
iompl«t« Htms 2H throuoh 2R and 
Hiini 3A through 3C. 



NoLJ I: 



MRT 2 (TO BE RECORDED AT COMPLETION OF REFERRAL PROCESS) _ _ 

G. Outcome of Referral Process (cont'd.) I. Previous Closure within Months: 

^ ACCEPTED 03 □-6-mo. bxt. Eval. (04) 

J FOR: 04 n-lS-mo. K\t. Lval. (06) 

05 VR Services (10) 

Compltti itwni2H through 2R. DO NOT 
COM^tEte ANY OF PART 3 AT THIS 




H. Disabling Condition (describe): 
1. Major 



Code . 



2. Secondary 



Code . 



Yes -OutuHiK' Re)«ab. [ ] 2 
Not Rehab. [ ] 3 
If Yes, Months Since l.js immure" 
J. Marital Status- 
K. Number of Dependent— 

L. Toul Number in Family — • 

M. Highest Grade Completed — 

N. Work Status 

0. Weekly Earnings $ 

P. Total Monthly Family Income 
(Including Earnings)- 

Q. Pubhc ^ Type 

Assistance Mo. Amt.-— $ 

' Time on PA 

R. Primary Source of Support 



PART 3 (TO BE RECORDED AT TIME OF CLOSURE) 



A. Federal Special Program Identification* 



None 



000 





Af R 






001 


002 



MMSIMAW rO 



004 



"joTO t020 





WIN 








SUM , 
















"040" 


100 


200 


400 ' 





B. Cost of Case Services 



(DoUars) 



C. 



I. All Services-Total 




2. Rehabihtation Facilities-Total 




3. Social Security Trust I'unds-Total 




4. Extended Lvaluation-Total 




Social Security I rust Fund Cases ONLY: 

I. Social Security Claim Type _ * 

U Claim Type Code 2, 3, or 4, enter Wage Earner's* 


Last Name First Name 


Initial 


Social Security Number 



J. Number of Months on Agency Rolls: 

1. In I xtended Evaluation (Sutus 04 or 06) 

2. From Acceptance to Closure (Statuses 10-24) 

3. In Training (StStus 18) 

4. Ready for or in Employment (Status 20 or 22) 

K. Outcome of E.xtended Evaluation or VR Services 

1 [j Closed from Ext. Eval. (Status 08); Reason 

2 □ Closed Rehabiliuted (Sutus 26) 

3 □ Closed Not Rehabiliuted (Sutus 28): Reason 

4 □ Closed Not Rehabiliuted (Status 30): Reason 

L. Services Provided: Indicate (x) if applicable 



Type of Service Provided 
or Arranged for by Agency 



With IWithout 
Cost Cost 
Only(I)|Qnly (2) 



t o Diagnostic andTvaluation 
\\_ Restoratio n (Phys. o r Menta l) 
12 



2. Check (x) if Administrativcf Costs Only □ 

D. Date Ext. Eval. Completed (if applicable) 

E. SSDI Status at Time of Closure 

F. Work Status ^ 



, College or U niversity 
T f Other Academic lbl6m./HS) 

r] 

A j Business School or College 

N Vocational School^ 
I 

N 

G 



i3 
14 
\S 

il 
18 

19 Maintenance 



On-theJob 



Persona l and Voc. A djustmcnj^ 



Miscellaneous 



G. Weekly Earnings-— 
IL Pubhc Assistance 

I. Occupation (title) 



4^ 



Type 

Mo. Amt.- 



2 0 Other Services ^ 

2 1 Services to O ther Fami ly Members 



With& 
Without 
Cost (3) 



V «/« 



U " 
g «^ 

BO 



GD 







M. Sute Agency Special Program Identification* 
None \1PU MIIC ,\VVCi ; 



. ^ Code 

•Thoso Items arc to bo coded. t lnchido;i counseling. guld«^"f e. and placement 

o ( 

ERIC 



000 ' 001 ' 002 ()(M 010 020 



(MO 



100 



200 [ 4()()_ 



C O CM 

-S^S! 

t> O ^ CM 

^ or*) 
C i - O 

C CM O 



< O V U. 

« o "5 
^sQo 

, . > o o 
U 5 u u 



SUM ; 



AGENCY COPY 
(PAGE 4) 



Date of Report Counselor's Signature 



Code 



CHAPTER 3 



TACKLING A PROGRAM EVALUATION PROBLEM: USING AVERAGliS 

• 

PE IS a tool which can help you strengthen your existing VR prognitn. 
This, in turn, can tell you and others how well the program is iunctioning. 

When you have properly evaluated one single aspect ol (he VR program, 
you will have information to either. 

1 . improve the program aspect, or 

2. account for the present practice. 

Either finding is highly acceptable. But - finding Ihe answer requires 
that you state the problem and objectives clearly and accurately. You 
have to look at the question raised and: 

1 . Define the problem. This can mean taking a vague criticism 
and pinningdown the real issue involved. Example: The State 
VR Director says clients stay in 00-02 statuses (referral-applicant) 
too long. Why does he think that? Let's assume that Agency X 
complained to him that a client remained in statuses 00-02 for six 
months and that it generally takes a long time before his clients 
are accepted or rejected by VR. Now you know the basis for the 
concern and can accurately define the problem. This.will help you 
decide your course of action and set the tone for your evaluation 
procedures. 

2: Look at the different ways the problem might be answered. 
Can you use currently available data? Is the problem related to a 
county and/or district, or does it have state-wide implications? 
How can the' problem be broken down into manageable segments? 

3. Decide on a systematic course of action. List the appro- 
priate steps in a sequential manner. Know what you want to do, 
when you want to do it, lind how it will be done. 

4. Gather appropriate information. Remember that the accuracy 
with which you collect your data will determine the accuracy of 
your results. If you can't get all fhe data you need, be sure you know 
how this limits the interpretation of your results. 

But don't let this stop you. 

5. Analyze the data and write the report. Make your report 
concise, and include appropriate data in a way that is readily 
understandable by others. 
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6. Make recommendations, based on your findings. 

7. I^'ollow up to see the results in terms of prognini clKinjie. 

Does this sound faniihar'^ It's the old decision-making process applied 
to PIv. To show you how it works, let's take a problem and follow 
through on what you might do. Remember that this is only one approach 
Ato the problem and is presented to serve as an example. After you have 
[gone through the procedure you might like to think through how you 
would have approached the problem in a different way. 

STATb DIRFCTOR- I think we are keeping clients in statuses 00-02 
too long. Find out what the situation is and let me know. 

PROGRAM EVALUATOR: What seems to be the problem? ' 

STATF: DIRECTOR: At the Lions Club luncheon yesterday, Mr. Smith - 
the director over at Agency -X - cornered me with this story about one 
of his referrals to VR that took six months before any action was taken on 
the appH,x:ation. He says that's not unusual, and he asked me what kind of 
a show we were running. I want to know what's going on! 

PROCiRAM HVALUATOR: Where is this guy from? 

STATI: 1)1 Kl. (TOR- He's over in Norris County. 

PROGRAM I'VALUATOR, I'll see what I can find out and let you know. 

So let's look at the task facing the program evaluator. 

PROBLEM To determine the length of time involved in processing referrals 
through statuses 00-02 and to look at this time in relation to referrals from 
all agencies, particularly Agency X. 

PROBLEM CONSIDERATIONS: 

1 . Agency X's director is wrong. His referrals don't really take 
longer, and he just picked out an exception. 

2. Agency X's director is right. His referrals take longer to 
process. 

3. Length of time for Agency X's referrals may be related to 
state-wide considerations, or may be a local problem. 

4. If Agency X's referrals generally take longer, why? Could it 
be, for example, that length of time in statuses 00-02 is related to 
the disability type of their referrals? 

11 
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WHAT DO WE NEED TO KNOW? 



I . How long does it take clients, on the average, to go through 
00-02 statjLises? 

7* Does the time vary for different referral sources regarding 
how long it takes clients to go through 00-02? 

3. Does the time vary in counties (offices, eJcO/oFclients to 
pass through 00-02? How does Agency X compare? 

4. How long does it take to process different disability categories 
thrpugh 00-02? 

SOLUTION: We know that clients pass from 00-02 to screening, and 
perhaps evaluation, to either acceptance or rejection (See Figure 1). 
A good indication of how long clients generally spend in 00-02 would be the 
average time of all clients in those statuses. 

Where do we get this information? Let's look at the currently available 
data on the R-300. Look at part 2, section E of the R-300 for the data 
on each client which shows months in 00-02. A simple tabulation of this 
figure (making sure all data is in months and fractions thereoO can be done 
for each district and/or county and the state (See questions 1 and 3 above). 

But let's go one step further. Part 1 , Section D of the R-300 contains a 
code for referral source (see question 2 above). Next, if we look at Part 
2, Section H, we see type of disability; and Parta , Section B yields us 
county data. 

Now we know where our data is going to come from. Our next decision 
is how we are going to handle the information. If you have your data on 
computer cards or tapes you simply tell the programmer you want a break- 
down as follows; 

1 . Average time spent in 00-02 for all clients for, say, the last 
reporting year. (This is the state average) 

■ 2. That average time broken down by. 

a. County (or district office) ' 

b. Referral source 

c. Disability category 

But for now, let's assume you have no programmer, are not knowledge- 
able' in statistics and are left on your own to figure out the answer. You 
have 10,000 completed R-300's to process by yourself, possibly with the 
help of one or two clerks. 
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First, 10,000 forms represent a lot of paper. You have to decide whether 
you can do all of them, or whether you will have to take a representative 
siimple. H you do not sample, you will have to process all 10,000 forms. 
If you do sample, the sampling must be correct or your results will be 
wrong. 

Let's assume you want to take a sample - say 1 0% or 1 ,000 of the 1 0,000 
R-300's is all you feel you should handle. You are now faced with the 
important decision of what 1 09r to use. If your sample is not representa- 
tive, then your results will be questionable. But, how do you get the proper 
sample? Since sampling can be highly technical, and much depends on the 
kind of sample you want, the best solution is to ask for help. 

You can go to your local university, Research and Training Center, or some 
other knowledgeable person and ask them to tell you which 1 ,000 of the 
1 0,000 R-300's you should pick and if t)iis is an adequate sample. Your 
authority will use a process for random selection and provide you with a 
list of numbers to use. The process is equivalent to putting all thelorms * 
in a hat, mixmg them up, drawing one, noting its number (e.g. social secu- 
nty number), putting it back in thorhat, mixing them up, and drawing a 
second number - and so on until 1 ,000 are drawn. That way, all forms 
have an equal chance of being selected. (You might end up using the 
same form twice.). 

The reason you should ask someone which forms to use is when you don't 
know hpw lo do a particular thing, ask for help. 

Let's assume that you have decided which 109^ of the forms (R-300) you 
will use based on the advice of your statistician -- and have arranged 
with your State Director to have access to these forms. This is your ran- 
dom sample. 

You are now ready to accumulate your data. Because this is painstaking 
and time-Lonsuming, you may want to make some decisions. For example, 
you might feel that the length of time in 00-02 is probably more directly 
related to county or district practices than to dpy^bility category. Thus, 
you would disregard disability category tcf^tfi down on the clerical effort. 
Admittedly, you may be wrong - but if you guess right, you can save 
considerable time. This is your decision. 

Assuming you are concentrating on county practices, proceed as follows 

1 . Put county names on individual sheets of paper (See Table 1 ). 

2. Subdivide each county into referring agencies. This is done 
because >ou want to see if the length of time is different for 
Agency X. 
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TABLE 1 




Boomer County 



Smith County 



Norris County 



Agency 



Time in 00-02 

Agency 2 Agency X 



3.2 mo. 


6.0 mo. 


1.3 mo. 


1.0 


0.2 


3.4 


2.0 


1.3 


3.5 


1.4 


2.2 


2.7 


1.8 


1.0 


2.5 


1.5 


1.4 * 


2.3 


1.9 


1.2 


3.7 


1 .9 Avg. 


1.6 Avg. 


2.8 Avg. 



State Averages for each Agency 
County Agency 1 



Norris 1 .9 

Smith 1.2 
Boomer 2.0 

State Average 1.8 
County Averages for all Agencies: 
Agency Boomer Co. 

Agency 1 2.0 
Agency 2 1.8 
Agency X 2.6 

County Average 2.3 

State Average for Time in 00-02: 

Boomer Cour 
Smith County 
Norris County 



Agency 2 

1.6 
1.9 
1.8 

1.7 



Agency X 

2.8 
2.3 
2.6 

2.5 



Smith Co. 

1.2 
1.9 
2.3 

1.6 



Norris Co. 

1.9 
1.6 
2.0 

2.0 



State Average 2.0 
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3. Transfer information from the R-300 to your chart (months 
in statuses 00-02 by county and referring agency). 

4. Compute your averages. You get an average by adding 
together all the months spent in statuses 00-02 and dividing 
that figure by the number of entries you made. 

a. Compute average for each referring agency in each county 
(See Table 1-a). 

b. Compute the state average for each agency (Table l-h). 

c. Compute average for each county by using the averages 
for all referring agencies in a particular county (Table 1 -c). 

d. Total up all counties and get average for state by dividing 
by number of counties (Table 1-d). 

Now you have some figures for comparison. Since you are not using a 
computer and are working with averages, you'll have to "eye-balP' the 
averages to get an idea of what is happening. 

Using our assunied figures from Table 1 , we know that in Norris County, 
Agency 1 referrals take 1 .9 months; Agency 2, 1 .6 months, and Agency 
X, 2.8 months. In Table 1-c, we see that the average time for all referrals 
in Norris County is 2.0. From Table 1 -b, we know that the average time 
for Agency X in allcounties is 2.5 months. Table 1-d shows an overall 
state average of 2.0 months in Statuses 00-02 for all referrals. 

The overall average time inX)0-02 for the state is"2 months. This answers 
question 1 under **What Do We Need To Know?" . Table 1-a and b show 
that referral agencies vary in time to process clients through 00-02. This 
answers question 2. Table 1-b and c answer question 3. We did not con- 
sider question 4. 

In our example. Agency X's director was right. His referrals take 2.5 
months on the average in the state, with a high 2.8 months in Norris 
County. We know this is higher than our state average of 2.0 months. 

'Our next step is to figure out why there is a difference for Agency X. 
Had our computations shown that Agency X was in line with other 
agencies on both a county and state basis, then we could have stopped 
here. 

We previously decided not to concentrate on disability category as 
influencing the length of time in 00-02. Let's assume that we now feel 
disability category might influence time in statuses 00-02. (We could 
have chosen other factors such as age or education of clients, etc.) Our 
next step is to repeat our data collection process and anatyze our data 
by disability category. 
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1 . Make sheets for each referral agency (similar to Table I ). 

2. Subdivide each-agency into disabihty codes. 

3. Transfer from R-300 time in 00-02 under appropriate 
disability code. 

4. Compute average time for each disability code for each 
referring agency. 

5. Compute percent of referrals by disability category for 
each agency. You can get this percentage by dividing the total 
number of R-300's for that agency into the total number in 
each disability category. 

6. Find average time for^ach disability code for all referring- 
agencies which will give you the state average for each disability 
code. 

Let's make another assumption - that clients in disability category "Y" 
spend a longer time in 00-02 on a state-wide basis than clients with other 
disabilities (e.g., 2.7 months). Let's say we found that 70% of Agency X's 
referrals on a state-wide basis fall into this category "Y". Now we know 
why Agency X's referrals take longer. It's related to the fact that most of 
their referrals are in disability dafegory Y, and that this category takes 
longer. We're dealing with state-wide figures so we know it is not related 
to county practices. We have identified a solution to our problem. Your 
• report to the State Director might be something like: 

1 . Our average time of clients in 00-02 is 2 months. 

2. When analyzed by county, there appears to be no differences 
in average time in 00-02 since counties vary from 1.6 to 2.3 months 

3. Wh?n we look at time in 00-02 by referral source, we find that 
Agency X referrals average 2.5 months state-wic|e with a high of 2.8 
months in Norris County. 

4. Further analysis shows that disability category Y takes an 
average Of 2.7 months 5tate-Wide. 

5 . Disability category Y represents 70% of Agency X's referrals. 

Therefore Agency X was rigf\t; his referrals take longer. This additional 
time appears to be due to disability category. 
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RECOMMENDATIONS. 

I Tell Agency X that his referrals do take longer on a state-wide 
basis and that it appears to be because of disability category of 
referrals. 

2., We should look at disability category Y to see why it takes 
longer and whether this time can be shortened. Since it is a state- 
wide situation,- we should look at our method of handling these 
cases. - 

> 

Like we said at the beginning of this chapter, an answ^requently raises 
other questions. Now you have to find out why disability category '*Y" 
takes longer. If there is a logical answer, then your State Director can also 
report this to Agency X. If not, you can look further. Investigate. 

If you, or some other dxjsignated person, do investigate the situation and 
institute new procedures for handling disability category Y, you can check 
these procedures' effectiveness by looking at data at the end of six months 
or a year. If you get a significant drop in the average time for processing 
disability category Y through 00-02, YOU HAVE DOCUMENTED PROGRAM 
IMPROVEMENT. 

You have also received other benefits. Now you can recbmmcnd to the 
State Director that he ask all supervisors to re-examine cases which stay 
in 00-02 longer than 2 months to see if there is a problem. If supervisors 
are able to cut down on the number of cases taking longer than 2 months 
m 00-02, then you are actually cutting down the state average from 2 
months to a lower figure. 

You can re-do your study using data from the next reporting period simply 
on client time in 00-02. If the average is less than 2 months, you\e 
documented program improvement- 

Now, let's assume you had given your problem to a programmer He would 
give you much the same results, the only difference being he would provide 
you with more statistical data and you d get it quicker. 

Other thing^you could have turned up in our httle exercise instead of our 
presumed solution could be: 

1 . The situatiOTi\mighl have been related to county practices 
rather than to disabHi^ty category. Maybe the office staff is new 
or understaffed, or there has been an extended illness - each possi- 
bility offers evidence for a proposed ^olution* 
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2. There were no differences and the Agency X Director was 
dead wrong. 

Ill either case, your State Director has data to support an answer and can 
plan corrective actions. 

Are you beginning to see the possibilities of what kinds of things you can 
investigate, using the ROOO? One of the biggest problems in this kind of 
PE is clerical help to assist you in wading through the paper. 
k 

Our advice is to use the corT\puter whenever possible, but if it's not avail- 
able, pencil power can do the job. The more knowledge you have about 
statistical procedures, the more you can do and the stronger your findings 
will be. But the important thing is to do what you can. 

Finally, let's look at some averages which you can compute from R-300's 
that might have evaluation potential. These are just to get you started 
thinking so you can devise your own combinations. 

1 . Profile of clients by state^and by district. For example: 

a. Average age. 

b. Average education- 

c. Average income at referral and closure, 
v B. Average time in statuses 00-02. 

Average tirne in statuses 0(T through 06. 

e. Average time in statuses to 08. 
^ Avgjage time in statuses to 26. 

Average Itim^ in statuses to 28. 
Average time in statuses to 30. 

f. Compare state and district profiles. 

2. I^umber of clients in disability category by state and district. 
Convert to percentage of total clients per state and district and 
compare. You might find different districts have different types 
f)f caseloads. Why? Is it referral source, etcv? 

3: Time-to type of closure (08, 26, 28, 30) by disability. 

-a. Do they vary? 

b. Compare district averages to state averages. 

c. Does one district have a higlier percentage of 26 closures 
for.certain disabilities? What does that district do that 
all districts might be doing? 

d. Are the percent of 26's for one disability higher than 
others? Why? 
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4 Average t»me to closure as 08. 26. 28. 30. by state and 
district for such things as: 

a age 

b. education 

c. referral source 

d. any other characteristics or category you are interested 
in. e.g. disabihty. 

If you never make it past Chapter 3 in this publication, you can still 
make a contribution to your VR program using the type of comparisons 
mentioned above. 

In our next chapter, we'll look at more problems, but won't go into as 
much detail and weMl examine other techniques. We're still sticking with 
R-300 data though. So, don't get up tight. 
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PROBLEMS 



L Compute the average of the following numbers: 10, 12, 36, 24, 9, 
18, 2L 13,22,41. (Answer: 20.6) • 

2. From the following information about 10 clients, compute the 
average time tO/closure by disability category and type of closure 

Disability Type of Time to 

Client No. Code Closure • Closure 



I 


520 


08 


1 4 months 


2 


640 


26 


1 2 months 


3 


532 


- 28 


23 months 


4 


119 


26 


3 months 


5 


510 


28 


9 months 


6 


614 


26 


1 0 months 


7 


146 


08 


5 months 


8 


630 


30 


3 months 


9 


530 


26 


17 months 


10 


131 


26 


7 months 



A typical answer: 

Average Time To Closure 

Disability 

Code Status 26 Statuses 08,28,30 

lOO's 5.0 5.0 

500's 17.0 15.3 

600's 11.0 3.0 

Total 11.0 7.7 



/ 



-19- 



CHAPTER 4 

ANOTHER PROGRAM EVALUATION PROBLEM: USING PERCENTAGES 



In the last chapter, we used averages to look at the time spent in statuses 
00-02. Now let's tackle another problem using percentages to help find 
the answer. 

ft 

A district supervisor becomes concerned when the number of 26 closures 
by one of his counselors far exceeds any other counselor in the district. 
Is Counselor Y really that much more productive than the others'^ Are . 
the other counselors not working up to par? Is Counselor Y successful 
because he takes only the easy cases? 

The district supervisor decides that the last question might hold the key 
to the problem. He wants to see if the old rehabilitation ''teeth and 
glasses" syndrome is in operation. Here's how he can find out: 

PROBLEM: To determine the disability categories of 26 closures by 
counselors in District A. 

PROBLEM CONSIDERATIONS: 

1 . What is the total number of 26 closures for each disability 
category for counselors in District A? 

2. Is Counselor Y's percentages of closures by disability category 
similar to the other counselors? 

3. If Counselor Y has more teeth and glasses closures, why? Is 
his caseload different in some way? 

^ ' 4. Does District A have a higher percentage of teeth and glasses 
'closures than other disabilities? 

'WHAT DO YOU NEED TO KNOW? 

1 . The percentage of clients closed in each disability category for 
the total district during the past reporting year (or last six months). 

2. The percentage of clients closed in each disability category for 
each counselor. 

SOLUTION: Teeth are included under the disability category of digestive 
disorders (660) and glasses under visual imj^airments other than blind (149) 
If you make a comparison of disil)ility categories by using percentages for 
the district and for each counselor, we can see what the district percentage 
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is and which counselors have larger or smaller percentages. In addition, 
if the percentages in these two categories are higher than other disabili- 
ties, you might want to investigate further. 

We go again to the R-300 to secure the information we need. Part 2, 
Section H gives the major and secondary disabling condition tor each 
client. You must decide whether you want to look at the secondary as 
well as the major disability. To do so, of course, will be more work but 
It will also give you more information. With the district code, and the 
counselor name and code on ea'ch R-300, you have all the information 
you need to proceed. 

1 . Make a sheet for each counselor in the district. 

2. Subdivide the sheets by disability categories. (You might 
want to add other pertinent information here such as time from 
referral to closure or case services expenditures.) 

3. Make a hatch mark (tally) for each case under the appro- 
priat^i disability for each counselor (see Table 2.). 

TABLE 2 



Counselor Z 
Counselor Y 
Counselor X 

Blind 

O'ther Visual 
Handicap 

Deaf 

Orthopedic 
Digestive 
Genito-urinary 

4. Count the hatch marks to find the totals in each disability 
category for each counselor, and the total case closures for each 
counselor. 

5. Compute the percentage each disability category represents 
in each counselor's caseload. (Total in disability category divided 
by total case closures). 



1 

11 

4-H+ 1 
irH+ 1111 
111 
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i6. Find the totals for each disability category for the district. 

7. Compute the percentage each disability category represents 
in the total district caseload. (Total in disability category divided 
by total district closures.) 

You can now place your percentages in a table like this: 



TABLE 3 





District A 


Counselor X 


Counselor Y 


Counselor Z 


Disability 


% 




- % 


% 


Blind 


1.9 


2.9 


1.1 


1.0 


Other Visual 










Impairments 


10.0 


10.6 


17.4 


5.7 


Deaf 


1.6 


1.4 


.8 


I.I 


Orthopedic 


14.5 


15.3 


10.2 


27.9 


Digestive 


22.a 


24.4 


31.2 


13.1 


Genito-Urinary. 


4.3 


'4.! 


3.2 


2.4 



Now you can compare by disalpility categories. What do you sec? From 
our hypothetical figures for District A, we learn that 10% of the closures 
fall into "other visual impairments" and 22% in digestive disorders; a total 
of 32% for both. Counselor X shows a total of 35%, Counselor Y 48.6% 
and Counselor Z 1 8.8% in these two categories. On the basis of this infor- 
ination, we can say that Counselor Y does indeed show a higher .percentage 
in these categories which include teeth and glasses cases than other coun- 
selors in the District. 

» 

With this information in hand, you as district supervisor should try to find 
out why Counselor Y shows such a high percentage. ' ^ 

Could it be related to the way in which cases are assigned to individual 
counselors? Is it just an accident that he has so mafiy of this particular 
type of case? Or, perhaps during this time period Counselor Y had a large 
number of long-term cases and the kinds of closures he has are different 
from the makeup of his total caseload. A quick case review can pick these 
things up. 

* • 

We also raised a question about District A's percentage of closures \f\ 
teeth and glasses cases. The figures show that disability categories con- 
taining these cases make up 32% of the closures. This seems to be a high 
percentage, so further investigation is needed here also. 
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You might want to compare these percentages with other districts in the 
state to see if they are different from District A. Or you might want to 
compare the percentages with the actual incidence of disabling conditions 
in the general population. These figures are available in Census data. 

What is important is that you investigate, seek new information when 
needed, and do all you can to get a true picture of what is happening. 
Then you can institute new procedures and programs to bring about change. 
A re-checking of closures six months later will telKyou if you have been 
successful in improving your program. 

You have now successfully provided data to cope with the problem you 
identified. But if you go back to Table 3, you will discover that you have 
found more than just the data to permit you to answer your question. 

Look at the percentages for Counselor Z. He has a small percentage of 
disability categories which include teeth and glasses closures, but look 
at the orthopedic cases. His closures for this disafbility are twice that for 
the total district. Does he prefer to work with these cases? Does he have 
a particular skill that makes him successful with persons with orthopedic 
disabilities? Just think of the possibilities for program improvement in 
terms of case assignment and counselor training, if either of.the above is 
true. 

At this point you might decide to go back to the R-300's and look at dis- 
ability categories and time clients spend on agency rolls for each counselor. 
And, while you are doing that, you might want to include case services 
expenditures. Are you beginning to get the idea? A little thought and 
planning and the possibilities fox meaningful PE are endless - and almost 
painless. 

^0. K., Mr. Counselor, we have been ignoring you. But you, too, can use 
PE to see how well you are doing. Why wait for the district supervisor 
to get on your back to get-this information? Check on yourself by using 
the procedures outlined above. You may learn some interesting things 
about yourself and the clients you serve! 
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! . Compute percentage of total number of clients in each disability 
category: 

Disability Code Number of Clients 

149 21 

399 5 

500 12 

660 17 

. 530 10" 

670 15 

(Answer: 149 - 26.2%; 399 - 6.2%; 500 - 15.0%; 660 - 21.2%; 530 - 12.5%; 
670 -18.8%) 

2. From the following information about 10 clients, compute the per- 
centage of type of closure by age and education: 







Yrs. of 


Type of 


Client No. 


Age 


Education 


Closure 


1 


22 


10 


26 * 


2 


39 


12 


08 


3 


49 


6 


28 


4 


19 


10 


28 




27 


• 12 


26 


6 


59 


9 


26 


7 £ 


■ 24 


8 


30 


8 


18 


7 


08 


9 


33 


13 


26 


10 


41 


,11 


26 



A Possible Solution: 



Percentage 'bX'C^lients 



Age and Education Status 26 Status 08. 28, 30 

Less than High School Grad. 

Under 30 years 20 60 

30 years and over 40 20 

High School Grad. and Over 

Under 30 years 20 - 

30 years and over 20 20 

100 100 
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CHAPTER 5 



USING THE COMPUTER 

There is a common belief that anyone who uses ttie computer tias either 
a degree in statistics or special courses in programming. 

Not so! 

Anyone can make use of the computer - if he has access to a computer 
programmer. If you are tired of paper and pencil computations and want 
to save time, let's jump into the world of computers. You can get quick 
results on your averages and/or^ercentage evaluations, or you can expand 
your efforts. \ 

The most important element in beginning to use the computer is to estab- 
lish a relationship with your programmer. For the most part, the studies 
you can do need only be limited by the ability of the two of you to commu-^ 
nicate effectively. This does not mean that you have to "speak his language" 
rather, it means you must be ^ble to clearly communicate to hmi just what 
you want the computer to do. Define your problem carefully and write 
down exactly what you want. 

Example: * 

PROBLEM: 

To determine the amount of case service money spent by type 
of disability. 

NEED: 

1. The various disability categories as shown (in code) on the 
R-300 form. Part 2, Section H. 

2. The average cost of all types of case ser^ costs as shown on 
R-300, Part 3, Section B. 

Your programmer will show you how to code your data so that it can be 
punched on computer cards. (Wc*1l go through an example later.) If you 
currently have R-300 data on computer can^ \ discs, or tapes, you don't 
have a problem However, there may be cases where the data have been 
recorded in a manner which is not readily usable for your particular type 
of evaluation project. In that case, the programmer may be able to repro- 
gram the data so it can be used. If such a rearrangement is not possible, 
you will have to provide him with the original R-300 data. 
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After he has placed the data on cards and has provided you with a computer 
print-out of the requested analysis, ask him lo explain the print-out to you. 
We wiM provide you with^some examples, but the format for print-outs varies 
so you sitbuld discuss th^ computer output with him. Ask him direct ques- 
tions and make notes directly on the print-out. 

We are now going to cover a PE problem the **eas5^" way and then look at 
. some of the possibilities of expanding your PE effort using more sophisticated 
statistical techniques. We will try to explain the purpose and uses of these 
techniques and won't refer to the way in which they arc derived because your 
computer will do thallor you. It is possible to work with your programmer 
if you know what you want and if you are wilHng to work with him to under- 
/^^'N(jpid the results. 

PROBLEM: The State |)irector wants to know the average time to closure 
(status 26) for the various counties/districts in the state. He then would like 
' to compare this average to the state and national average. (We will assume 
the national average to be 15 months.) 

SOLUTION j • (Paper and pencil method) 

1 . Get all the R-300's closed as 26, as shown in Part 3, Section 
^ K.2, for a reporting period; e.g., a year. (You can do this by 

sampling if you wish, as we did in Chapter 3,) . 

2. List each county/district on a sheet of paper (R-JUU, Part 1 , 
Section B). Post time to closure for all cases under aprropnate 
county, using the totals of: * ' 

a. Months in statuses 00-02 (Part 2, Section E of ^R-300). 

b. Number of months on agency rolls (Part 3, Section-J, 1-2), 
Note: You may or may not want to use (a) above, but if 
you don't, you should state that in your summary of results. 



3. Compute the average for each county. 

4. Add up all the county totals and divide by the total number 
of cases to get the state average. 

You now have the information for your comparison - average time to closure 
(status 26) by (1) county and (2) total state. 

What we have done so far is a repeat of what was done in Chapter 3. Let's 
now take a look at how this same problem can be handled on the computer. 

r 

» 
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^ SOLUTION 2' (Expanded solution) 

, I . Consult withlyour programmer First, discuss with him the 
' J data you need and rtje comparisons that must be made. He can 

provide you with spreM^bwts (See Illustration 1 ) with 80 columns 
on it. These columns correspon^vto the 80 columns on ;i computer 
card. More than likely, heMl ask yVub put the county code in 
three columns since the county code is a 3-digit number. Just to 
identify the R'300 forms used, you will also place the social security 
number in the appr6priate number of columns - 9. Then, heMl ask 
you to put the total number of months to closure in three other 
columns; e.g., 020 for 2 months, 025 for ? 5 months, 103 lor 10 3 
months and so on. 



Note: If you have decided you want additional comparisons such 
as a breakdown by disability category, you will have to add that 
information on the spread sheet at this time. 

2. Tell your programmer that you would like to have the average 
time to closure for each disability category by county and by state. 
(This is exactly what you would have done by pencil and paper.) 

3. T^ll your programmer that you would like to find out if any 
of these county averages for the various disability categories are 

' far enough above or below the total state average for that disability 
category to show that the two averages are statistically different 
from each other. In other words, is the average for a particular 
disability category in county/district statistically different from the 
state average? ^ 

4. Your programmer will take your sheets, have them punched on 
cards, and conduct the analyses for you. You will get back a series 
of computer print-ouls. Ask the programmer to go over these print- 
outs carefully with you, showing you where your answer to each 
question is. 

/ 

We would like now to give you some background to help you when you talk 
wtth the programmer. 

The programmer will tell the computer to add up the totals by disability category 
for each county and get the average. He will then tell the computer to add up 
ihe ttrt'als of all counties for each disability category and get the state average 
for that category - just what we did in Solution 1. 

Then, the programmer will tell the computer to compare the disability category 
average for each count/with the total state average for that disability category 
to see if it is statistically different. 
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He may use what is called a *'t-test'' which will individually compare each 
county average with the state average, one at a time. In this case, you will 
get a print-out for each counjy with a *'t'^ value and a ^^probability leveP' 
(See Illustration 2). 

Then again, he may tell the computer to compare all means at once, using 
a technique called '^Analysis of Variance'' which permits the computer to con- 
sider two or more groups (in this case, all counties and the state) at the same 
time. This print-out is harder to read. 

It will give you an 'T" value (named 3fter Fisher ^ho figured out how to do 
It) and a "probability level" plus some other comparisons. These comparisons 
are really what you are interested in, for they will be the ones that compare 
county/district averages to state averages. (See Illustration 3) 

Now let's take a look at an explanation of what you see on Illustrations 2 and 



First, the only thing you really need to know about and "t" is that the 
larger the number is, the more likely you are to have a difference among or 
. ' between your averages. If you look in Illustration 2, for example, you see 
t = 1.056, and the probability level is .293. It's the probability level that 
you'll be most concerned with. 

Now when you look at your computer print-out, the probability level will 
tell you exactly how many chances in 1 00 you have of having a difference 
between two averages. For purposes of your evaluations, we suggest you set 
a probability level of .05, which means you should have only Scfignces in 100 
of being wrong. You, therefore, can be 95 percent sure that .your averages are 
different. 

Let's look at an example. In Illustration 2, you will note that^^this probability 
level has been calculated by the computer^abe .293, meaning you would have 
morej3f^ chance of being wrong than.5 in 100. We now know that the averages 
are ^t statistically different. So, when ypi^look at probability levels, any num- 
h&rXhai is less than OS (like .0499, .o;320, .026^, cic4 lets us know that we have 
^^significance above our .05 level and tKere is a difference between averages. On 
the other hand, numbers that read .OS'Ol , .0629,^2930, .3645, etc., are all non- 
significant; meaning that there is more than a 5% chance that the averages are . 
not really different. Therefore there fs no difference^between the averages. 

Probability levels are treated the same for all tests - both F and t's. However, 
the F comparisons get more involved because of the different ways the program- 
mer can compare individual county/district averages to the state average. After 
the F comparison he may use t tests or other means of comparisons. You should 
get him to circle the significant comparisons on the print-out and label them for 
you. 
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Takc notes and write his explanations right on the print-out for future refer- 
ence. For example, if he says the- county/district coded as 034 is signifi- 
cantly different from the state average for disability category "X". circle 
that result (since you coded the county/district, you know that the nunv 
bcr 034 represents a particular county/district, say Norris County). 

Now, let's look at some of the things you might do with your comparisons. 
Say Norris County's average time to closure is higher on a particular 
disability category than the state. You might want to recommend case 
review to find out why Norris County takes longer to close in status 26. 
On the other hand, Boomer County might take significantly less time, 
in which case it might be worthwhile to see what Boomer County is doing 
that might be applicable across the state. See Table 4 for a comparison of 
three hypothetical counties. What we have done here is make a three-column 
table. We have listed our counties with their averages for disability category 
X. We have placed our t-values in the last column and have noted,with an 
asterisk, those that were significantly different at the .05 level or better. 
Also we have noted at the bottom of the table just what level we have cho- 
sen. 

\ 

You will note that Boomer takes significantly less time to closure than our 
state average, while Norris takes significantly longer. However, Smith 
county varied, but not sufficiently to'be significant (13.3 was close enough 
to 14 to be regarded as being the same). It might appear on the surface 
that eyeballing is as good as using the probability level method. However, 
the computer considers such things as number of cases you are looking at. 
Fifteen might be different from 1 4 for one disability category, but not 
different for another, depending on just how the length of time varies and 
how rnany people are in that disability category. 

There is almost no end to the amount of data you can put in a computer 
and the ways it can be compared. Can you see categories on the R-300 
that you would like to see compared for county/district: education? case 
service money spent?^ disability, etc.? 

Alf of these comparisons can be done for you, following -tlhe procedures 
we outlined above. It your computer programmer diff^s with our recom- 
mendations, follow his advice, for computers and computer programmers 
differ, and he will know what programs to use and the limits^of these pro- 
grams. 
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TABLE 4 

Averages of Selected Counties: Time to 26 Closure by Disability 
Category X 

County Average t-value 

Norris - 15.2* ' 2.086* 

Smith. . 13.3 . 1.942 

Boomer 5.0* 13.845* 



♦significant at .05 level 
State Average = 1 4 months 



-34- 



WHAT HAVO \VT- LHARNED IN THIS CHAPTER'' 

1 . That two groups can be compared for differences using the t-test, 
and that two or nx)re groups can be compared using Analysis of Variance. 

2. That differences show up in the form of the probabihty'that the dif- 
ferences are significant. 

3. That significance is determined by the probabihty level which you 
choose, usually the .05 level. 

4. That data not on cards or tapes are placed op spread sheets in a 
particular formal so they can be punched for the computer. 

5. That yo»'r best friend is your progr<immer when you use a computer. 

We do net expect that you will have grasped all of the concepts in this 
chapter. However, after you have worked in PE for a while, inore of 
these techniques will become clear. We have presented these concepts 
just in case you have a little background in statistics or math. If you got 
the general idea, great. Perhaps an explanation from your programmer 
or other knowledgeable person will t?elp clarify things further. 
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CHAPTER 6 
REPORTING " . 

♦ 



Let's say you now have a completed PE report, analyzing some part 
' Of your program. What do you do now? 

We're suggesting that a large part of your job still lies ahead. A PE 
report will never accomplish anything unless it is used, unless it gets 
heard in the places where decisions are made in your agency? If action 
is not taken in terms of program improvement, then why have done PE 
in the first place! 

One way to make sure that your work will get appropriate attention is 
to have a conference with the State Director and get clear understanding 
as to the types of data he thinks are needed for the agency. There is a 
lot of research that indicates that when information meets a felt need, 
it is more readily utilized and acted upon. ^ ' * 

Another suggestion we haye regarding your reports is that there should be 
some formal procedure for seeing that the reports get proper review and 
are giveij serious consideration. 

One such procedure is offered for your consideration. It might go some- 
thing like this: 

The results of an evaluation should be presented in a Written report that 
clearly states its conclusions, and data upon which the conclusions were 
based. Recommendations should be included where indicated. 

The initial report would be sent to the agency administrator or his desig- 
nee. Copies of the report would then leave the evaluator's desk earmarked 
for review by agency personnel directly responsible for the program aspect 
under evaluation. A written response should be expected from program 
personnel within a reasonable or designated time period. 

The program personnel should respond to the report and would have an 
obligation to make recommendations relative to the need for changes and 
additions within the program. This response should be sent to the admin- 
istrator's office for policy and administrative consideration. 
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Perhaps the most important part of this procedure is the responsibility 
how facing the office of the agency administrators. 

It is the responsibility of the program evaluator to make the state agency 
director fully aware of the program evaluation findings. Tlie director's 
response to the findings of an evaluation report could be the single most 
important factor in its implementation. 

Once a progrkn administrator has made some decision in response to the 
findings of program evaluation, there is^still a need to monitor the imple- 
mentation of that decision. Too often decisions are made but the follow- 
through on implementation is inadequate. 

The responsibility for monitoring should be explicitly assigned by the 
agency director along with the requiremen^t for a report back to his office 
within a set time period. ' 



Schematically, a procedure such as the one elabohtted above would look 
like this: 




ACIBNC Y AND ADMINISTRATION CYCLB 



EVALUATION 





REPORT 



EVALUATE 



(CRITIQUE) 
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Why don't you see if you can getstmie understanding as to what would 
be an appropriate procedure and probably more important, what would 
^ be an acceptable one. Then go ahead and start your evaluations confident 
at least that they'll be given some consideration. 

In the rest of this chapter, we'll offer some reports you might do as part 
of your routine operation. 

A^I^ALLY 

1 . Profile of clients closed as 26's, compared by state and dis- 
trict, with any variations on such items as: 



a. Average age 

b. Average education 

c. Average income prior to VR services 

d. Average income after closure 

.e. Percent of caseload by disability category 

f. Percent of caseload by referring agency 

g. Average time to closure 

h. Any other items you wish to consider, like percent of 
caseload by sex, etc.-- 

2. Same profile as in Number 1 for 28's and 30's by state and 
district. 

3. Profile of clients by disability category exiting at status 08, 
their sex, age, referral <!oiirce, etc. Do they differ from statuses 
26, 28 and 30 on some characteristics? 

4. Case service expenditures by disability category, by state 
and district. 

5. Case service expenditures by months to closure (26, 28, 30 
and 08, if applicable) by state and district. 

6. Average months in statuses 04 and 06 by type of closure 
and/or disability category) by state and distri^. 

7. Average months in statuses 1 0-24 or its sub-statuses by type 
of closure and/or disability category, by state and district. 

8. Similar reports as shown in 1-7 for Federal special programs. 

9. Time in statuses 00-02 by disability, state and district. 
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1 0. Number of clients dosed who had been previously rehabil- 
itated by disability, district and state. 

1 1. Number of closures, 08, 26, 28 and 30 compared to number 
of cases accepted for VR services and how we compare to.national 
or state figures. 

12. Any other aspects of your program you feel would be valuable 
to look at. 

QUARTERLY 

1 . Do any of the above every three (3) months to get quarterly 
comparisons (gets at fluctuations throughout the year). 

2. Case service expenditures by quarter by district to see if 
funds are being budgeted properly. 

MONTHLY 

1 . Case movement - how many clients move among the various 
statuses. Make graphs for each month for each status. See how 
they vary. This may help smooth out case flow problems by pro-- 
viding data on case movement and then you could compare dis- ' 
tricts tb thci state average. 

Before we close, we'd like to stress two things. 

First, be-aware that many aspects of a rehabilitation program are "special" 
in that they serve unique kinds of clients. 

Findings, for example, from a study of a penitentiary unit, mental hospital 
or an alcoholism rehabilitation unit will necessarily be quite different than 
data gathered from the general field program which delivers diverse services , 
to varied target populations. So watch out for differences that on the surface 
seem meaningful, yet only reflect a difference in the type of client served. 
Such data can be statistically different, yet be useless for meaningful program 
reform. 

Secondly, Mj. District Supervisor and Mr. Counselor, don't be misled into 
believing thatTE is only for the "Chiefs" and the "Higher-ups." The con- 
cepts presented here are as applicable to you as to them. 

Supervisors can keep track of their units using the very same techniques 
proposed here. Additionally, counselors can monitor their own caseload 
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using exactly tlxe same procedures. In fact, Mr. Supervisor, if every coun- 
selor kept a ''running" list of these data, it#3uld be a simple task to get 
district or unit figures that would be meaningful for local program reform. 

So remember! PE is not discriminatory! It is everybody's business! 
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CHAPTER 7 
DATA NOT CURRENTLY AVAILABLE 

Lots of times, very meaningful questions pop up in VR operations to 
which there are no immediate answers because the data aren't available 
to help you. ^ 

f 

The hardest evaluation problems are sometimes. those for which current 
data are not available. 

A primary problem in these kinds of stu^iies is in precisely what you want 
to measure. For example, a perennial problem that faces rehabilitation 
is **Who are the best counselors"? 

A most appropriate resppnse to that question is *'What do you mean by 
* besf ? The ones who get the most 26 closures? Maybe - maybe not. 
The ones who do the best counseling job? That's a real tough one. It 
requires that you define exactly what you want to check. Is it actual 
time spent with clients? Or could it be one of the many possible defini- 
tions of best, good, effective, etc.? 

So, when you tread into this domain of evaluation, be prepared to en- * 
counter many problems that require techniques of a somewhat sophisti- 
cated nature. Maybe these kinds of endeavors could be postponed - at 
least until you feel more comfortable with the procedures outlined in 
Chapters 3 and 4. 

Therf again, maybe you could try! You be the judge! 

The rest of this chapter will give you three examples of the types of 
evaluation that you pould do that fall into this category. 

1 . Survey of percent of time counselors spend in various duties: 

Dictation, writing 
Reading 

Attending conferences 
Fiscal tasks 
Statistrcailasks' 
Travel 

Public relations 
Supervision 
Referrals 




a. 
b. 
c. 
d. 
e. 
f. 

g. 
h. 
i. 
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j. Determining eligibility 

k. Counseling 

I. Planning client services 

m Placement and follow-up 

Then you'd know precisely what your counselors are doing, how they're 
spending their time.- For example, it might come out that more time is 
spent filling out forms than in counseling-or placement activities. Based 
on these data, you could then answer questions such as: Is this desirable? 
If not, what can be done to change it? Either way, you have data available 
allowing you to make decisions regarding the need for improvements. 

2. Survey of counselor expertise; e.g., what kinds of clients do 
you prefer to work with? Use a ratineibrm which asks counselors 
to rate client preferences by (a) dis^tjuity, (b) socio-economic level, 
(c) sex, (d) referral source, etc. ^ 

You might find counselor preferences which would assist you in assigning 
caseloads. Maybe Counselor A prefers working with mental disabilities 
as opposed to physical disabilities. Chances are, the counselor prefers 
doing what he does best - and what better way to assign clients insofar ^ 
as possible. 

3. Follow-up survey of closed and terminated cases from a 
rehabilitation facility or special program. You could ask questions 
such as: 



a. Had they worked? 

b. How long had they worked? 

c. Was the job in an area closely related to their training? 
' d< How much money did they make? 

^ t. What are the client's perceptiOTis^of the worth of 
rehabHlitatioB services provided? 

The above are only ^suggestive. ^e'?e sure you could come up with four 
er five of your own. At any rate, try them - if you feel comfortable 
with them. 

BUT REMEMBER! 



Stick to evaluation areas that you feel competent to deal with and branch 
out as your skills branch out. Don't hesitate to discuss your study with 
resource people, especially your programmer. They'll tell you how you 
can best collect data. 



